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Abstract 



We present the application of Principal Component Analysis for 
data acquired during the design of a natural gesture interface. We 
investigate the concept of an eigengesture for motion capture hand 
gesture data and present the visualisation of principal components 
obtained in the course of conducted experiments. We also show the 
influence of dimensionality reduction on reconstructed gesture data 
quality. 



Human-computer interface (HCI) which uses gestures promises to make 
certain forms of user interfaces more effective and subjectively enjoy- 
able. One of important problems in creating such interface is the 
selection of gestures to recognize in the system. It has been noted [ I '] 
that choosing gestures that are perceived by users as natural is one of 
decisive factors in interface and interaction performance. At the same 
time, a large amount of research is focused on fixed movements geared 
towards efficiency of recognition, not interaction [13]. 

We view the analysis of natural gestures as a prerequisite of con- 
structing an effective gestural HCI. As a tool for this task, it is natural 
to use Principal Component Analysis (PCA) [7]. PCA has been suc- 
cessfully applied for analysis and feature extraction i.e. of faces (the 
famous 'eigenface' approach [ ]). For human motion, PCA has been 
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found to be a useful tool for dimensionality reduction (see i.e. [1 I]). 
Eigengestures appear in a number of publications, i.e. [ ], where 
they are used as input for motion predictor. In [15] they are used 
for synthesis of additional training data for HMM. In [3] eigengesture 
projection is used for real-time classification. We argue, however, that 
the eigenvectors of human gestures-especially hand gestures-should be 
investigated beyond the effect they have in improving data processing 
{i.e. classification score); the structure of the decomposition may lead 
to important clues for data characteristics, as it has been the case for 
images [6]. To the best of authors' knowledge, this is a still a research 
field with limited number of contributions: in [16] eigen-decomposition 
of 2D gesture images is only pictured without discussion, whereas in 
[17] a basic analysis is done only for whole body gestures; main eigen- 
vector are identified with deictic (pointing) gestures. 

The main contribution of this work is application of PCA to anal- 
ysis of the data representing human hand gestures obtained using mo- 
tion capture glove. We show the influence of dimensionality reduction 
on reconstructed signal quality. We use the notion of eigengesture to 
the collected data in order to visualize main features of natural human 
gestures. 

This article is organized as follows. Section 2 presents the exper- 
iment methodology; the sample set of gestures, acquisition methods, 
participants and procedure. Section 3 details the application of PCA 
to motion capture gesture data. Section 4 presents discussion the 
computed principal components. Section 5 presents visualization of 
eigengestures. Last section presents concluding remarks. 

2 Method 

For our experiment, we used base of 22 different type of gestures, each 
type represented by 20 instances — 4 people performing the gestures, 
each of them made the gesture 5 times (three with normal speed, then 
one fast following with one slow execution). The gestures are detailed 
in table 1. For discussion on gesture choice the reader is referred to [ ]. 

The gestures were recorded with DG5VHand motion capture glove [J], 
containing 5 finger bend sensors (resistance type), and three-axis ac- 
celerometer producing three acceleration and two orientation readings. 
Sampling frequency was approximately 33 Hz. 

The participants for the experiments were chosen from the employ- 
ees of Institute of Theoretical and Applied Informatics of the Polish 
Academy of Sciences. Four males were instructed which gestures were 
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Table 1: The gesture prepared with the proposed methodology 





Name 


Class" 


Motion^ 


Comments 


1 


A-OK 


symbolic 


F 


common 'okay' gesture 


2 


Walking 


iconic 


TF 


fingers depict a walking person 


3 


Cutting 


iconic 


F 


fingers portrait cutting a sheet of paper 


4 


Shove away 


iconic 


T 


hand shoves away imaginary object 


5 


Point at self 


deictic 


RF 


finger points at the user 


6 


Thumbs up 


symbolic 


RF 


classic 'thumbs up' gesture 


7 


Crazy 


symbolic 


TRF 


symbolizes 'a crazy person' 


8 


Knocking 


iconic 


RF 


finger in knocking motion 


9 


Cutthroat 


symbolic 


TR 


common taunting gesture 


10 


Money 


symbolic 


F 


popular 'money' sign 


11 


Thumbs down 


symbolic 


RF 


classic 'thumbs down' gesture 


12 


Doubting 


symbolic 


F 


popular (Polish?) fiippant 'I doubt' 


13 


Continue 


iconic'^ 


R 


circular hand motion 'continue', 'go on' 


14 


Speaking 


iconic 


F 


hand portraits a speaking mouth 


15 


Hello 


symbolic*^ 


R 


greeting gesture, waving hand motion 


16 


Crasping 


manipulative 


TF 


grasping an object 


17 


Scaling 


manipulative 


F 


finger movement depicts size change 


18 


Rotating 


manipulative 


R 


hand rotation depicts object rotation 


19 


Come here 


symbolic*^ 


F 


fingers waving; 'come here' 


20 


Telephone 


symbolic 


TRF 


popular 'phone' depiction 


21 


Co away 


symbolic'^ 


F 


fingers waving; 'go away' 


22 


Relocate 


deictic 


TF 


'put that there' 



° We use the terms 'symbolic', 'deictic', and 'iconic' based on McNeill & Levy [!)] classification, 
supplemented with a category of 'manipulative' gestures (following [11]) 

^ Significant motion components: T-hand translation, R-hand rotation, F-individual finger 
movement 

This gesture is usually accompanied with a specific object (deictic) reference 
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going to be performed in the experiments and were given instructions 
how the gestures should be performed. A training session was con- 
ducted before the experiment. 

During the experiment, each participant was sitting at the table 
with the motion capture glove on his right hand. Before the start of 
the experiment, the hand of the participant was placed on the table in a 
fixed initial position. At the command given by the operator sitting in 
front of the participant, the participant performed the gestures. Each 
gesture was performed three times at the natural pace. Additionally, 
each gesture was made once at a rapid pace and once at a slow pace. 
Gestures number 2, 3, 7, 8, 10, 12, 13, 14, 15, 17, 18, 19 are periodical 
and in their case the single performance consisted of three periods. 
The operator decided about the end of data acquisition. 

Input data consist of sequences of vectors gt„ G H^^,n G {l,Ni} 
which are state vectors of the measurement device registered in sub- 
sequent moments tn of time. The time difference — t„ is almost 
constant and approximately 30 ms. Each registered gesture forms a 
matrix d G M]\f. ^io{R). Acquisition time for every gesture is differ- 
ent, therefore the number of samples Ni depends on the sample. We 
acquired K = 22 different gestures, which are repeated L = 20 times. 

3 Data processing 

Our chosen statistical tool was Principal Component Analysis (PCA). 
It has been successfully applied in the domain of signal processing to 
various datasets such as: human faces [J2], mesh animation [2]. 

3.1 Principal Component Analysis 

For the sake of consistency we start by recalling basic facts concerning 
Singular Value Decomposition (SVD) [5] and Principal Component 
Analysis (PCA) [7]. 

Let A £ Mm^„ has rank k < m. Then there exist orthogonal 
matrices U € Mm and V G M„ such that A = UT,V'^ . The matrix 
S = {(Tij} G Mm,n is such that aij = 0, for i ^ j, and an > (T22 > 
■ ■ ■>crkk> crk+i,k+i = ■ ■ ■ = CTqq = 0, with q = min(m, n). 

The numbers an = ai are called singular values^ i.e. non-negative 
square roots of the eigenvalues of AA^ . The columns of U are eigen- 
vectors of AA^ and the columns of V are eigenvectors of A^A. 

Principal Component Analysis allows us to convert a set of obser- 
vations of correlated variables into the so-called principal components, 
i.e. a set of values of uncorrelated variables. 
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Formally, the i-tli principal component is the i-th column vector of 
the matrix V-^i x an obtained as a SVD of the data matrix. In order 
to perform PCA on the data acquired in different units, the data need 
to be unified to a common units. In our case, the initial vector of data 
is transformed by the studentisation, i.e. by subtracting the empirical 
mean and dividing by the empirical standard deviation. 

3.2 Organisation of data 

As the input of the algorithm we have K x L matrices Gj. Each matrix 
represents a single realisation of a gesture. In order to perform PCA, 
the data are transformed in the following way: 

1. Re-sampling: for every signal indexed by s £ {l,...,^ = 10}: 
Gt„,s — G[, ^, where tn indexes time samples of the gesture as 
acquired from the capture device, t'„ G {1,...,N = 20} using 
linear interpolation. 

2. Arranging into the tensor: 

where k G {1, . . . , if = 22} denotes number of the gesture type 
and I G {1, . . . ,L = 10} denotes individual realisation of a ges- 
ture. 

3. Double integration of signal from accelerometers to transform 
acceleration into position variable. 

4. Centring and normalisation: for every signal s: T'^i - s = {Tk,i,:,s~ 

The data are arranged into a matrix ^(fc,z),(t„,s) = Tj^u s "^hose 
columns consists of vectorised distinct realisations of gestures. Such a 
matrix is then feed into SVD algorithm. 

A sample of our data is visualised in Fig. 1 a) which presents the re- 
sampled, centred and rescaled second realisation of the Cutting gesture 
described in our data tensor by sub-matrix T^2 -. :■ 

4 Application of PCA to data explora- 
tion 

One of the typical applications of PCA to the analysis of the data 
obtained from the experiment is to reduce their dimensionality. Fig. 2 
shows mean quality of the approximation of the original dataset in 
function of the number of principal components used to reconstruct 
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Figure 1: A sample of the gestures dataset. The data are normahsed and 
centred. Single realisation of Cutting gesture. Upper plot bending of fingers: 
T — thumb, / — index, M — middle, R — ring, L — little; lower plot: 
dashed line — palm roll, dotted line — palm pitch, X Y Z — palm position 
in space, a) original data, b) approximation reconstructed using only 20 first 
principal components. 



the dataset. The distance in the Figure is scaled so that the approx- 
imation using only the first principal component gives 1. It can be 
easily seen that the dataset can be efficiently approximated using low 
rank approximation. 

A comparison of original data sample vs its low rank approximation 
is shown In Fig. 1, sub-plot a) shows original data for Cutting gesture 
and sub-plot b) shows the same data reconstructed using only first 20 
principal components. 



5 Visualization of eigengestures 

The coordinates in which the eigengestures are obtained are artificial. 
To create a visualisation one needs to change the coordinates to suited 
for the hand presentation model. 

The change of coordinates is obtained by affine transformation act- 
ing independently on each dimension (sensor data). The parameters 
of these transformations (scales and translations) are obtained in the 
following way: 

• The scale factor for each sensor (dimension) is a quotient of 0.05 
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Figure 2: Relative Euclidean distance between the dataset and its approxi- 
mation obtained using first n principal components. 

- 0.95 quantile dispersion of tliis sensor data and tlie dispersion 
of the sensor in the eigengesture. 

• The translation is calculated in such way that each visualised 
eigengesture has unified starting position. 

In Fig. 3 the first two eigengestures (principal components) are 
shown. The first eigengesture looks very natural and resembles ges- 
tures commonly used by humans in process of communication. We 
found that higher eigengestures do not look very natural especially 
because of the negative values of the finger bends. Due to orthogonal- 
ity of left singular vectors obtained from SVD it can not be expected 
that eigengestures will be similar to natural gestures performed by 
humans. 

We observed that time plots of eigengestures around number 100 
and higher are very noisy. 

6 Conclusions and future work 

In this work our goal was to explore the space of human gestures using 
Principal Component Analysis. Visualisation of eigengestures is a tool 
that allows us to understand better the dataset we acquired during 
the experiment. We have shown that natural human gestures acquired 
with use of motion caption device can be efficiently approximated using 
50 to 100 coefficients. Additionally we have identified the principal 
component of the gestures dataset. 

Future work will consists of application of the obtained results to 
analysis of quality of gesture recognition. We will compare PCA with 
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Figure 3: Visualization of two first eigengestures (principal components). On 
top: normalized and centred plots of signals in time. Upper plot bending of 
fingers: T — tfiumb, / — index, M — middle, R — ring, L — little; lower 
plot: dashed line — palm roll, dotted line — palm pitch, X Y Z — palm 
position in space. At the bottom: shapes of hands in selected time moments. 
View is from the perspective of a person performing the gesture. For the 
sake of the clarity of the picture space position of the palm is omitted. 
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Higher Order Singular Value Decomposition [c ] as data dimensionality 
reduction techniques. 
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